Skip to content

Files features#619

Merged
chvp merged 42 commits intomasterfrom
files-features
Mar 9, 2026
Merged

Files features#619
chvp merged 42 commits intomasterfrom
files-features

Conversation

@niknetniko
Copy link
Member

Branch with all file features, so we can merge in one go (and revert in one go if needed)

…ges"

This reverts commit 16ce3cf, reversing
changes made to 3329a84.
Cattrs works by passing the value it is structuring into the default
constructor of the target type, so it supports Python's coercions.

In this case, consider the previous version of TextData:

class TextData:
  data: str
  type: "text" | "file"

When a test suite containing {'type': 'text', 'data': 500} is
structured, cattrs will attempt to structute `data` by doing `str(500)`.
This is legal, so cattrs happily accepts it.

The new definition, however, is as follows:

class TextData:
  content: ContentPath | str
  path: str | None = None

When the decorator runs and returns the value from data as an int,
cattrs will take over, trying to structure it into `ContentPath | str`.
TESTed has a custom union structuring function to allow structuring
into any union. One thing it does not do, however, is the data coercion.

The union structuring requires exact types, which is why the code failed
and the old test suite no longer works.

To fix this, we now explicitly coerce the legacy data to a string inside
the fallback converter before it reaches the strict union hook.
@niknetniko niknetniko marked this pull request as ready for review March 5, 2026 19:58
@chvp chvp self-requested a review March 6, 2026 12:14
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces end-to-end “file features” across the TESTed ecosystem (schema/DSL parsing, testsuite model, planning/execution, and oracle evaluation), enabling input files, stdin-as-file, and multi-file output comparisons.

Changes:

  • Refactors suite data structures to model text/file content via TextData(content, path) and introduces multi-file FileOutputChannel(files=[...]).
  • Extends DSL + JSON schema to support stdin as file data, input_files, output_files, and !path references.
  • Updates planning/execution/evaluation/oracles to create/copy dynamic files and to report per-file oracle results.

Reviewed changes

Copilot reviewed 70 out of 70 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
tests/tested-draft7.json Extends schema test types with path.
tests/test_testsuite_legacy.py Adds legacy-format parsing regression tests.
tests/test_suite.py Updates tests to content/files[] model.
tests/test_planning_sorting.py Adds tests for dynamic-file collection/sorting/dedup.
tests/test_planning.py Adds planning tests for file/stdin/output conflicts and splits.
tests/test_parsing.py Adds tests for new parsing decorators and converter hooks.
tests/test_oracles_programmed.py Adds programmed-oracle coverage for output files.
tests/test_oracles_builtin.py Updates built-in oracle tests for new file oracle + multi-file.
tests/test_io_exercises.py Adds IO exercise plans covering stdin + dynamic files.
tests/test_functionality.py Updates stdin tests to TextData(content=...).
tests/test_file_linking.py Adds tests for file-link messages and readable input linking.
tests/test_dsl_yaml.py Updates DSL YAML tests and adds stdin/input_files cases.
tests/test_dsl_legacy.py Adds DSL legacy compatibility coverage for files/output files.
tests/test_collector.py Ensures per-file NOT_EXECUTED reporting on termination.
tests/exercises/time-2-code/solution/solution.ts New fixture solution for file I/O exercise.
tests/exercises/time-2-code/solution/solution.sh New fixture solution for file I/O exercise.
tests/exercises/time-2-code/solution/solution.py New fixture solution for file I/O exercise.
tests/exercises/time-2-code/solution/solution.kt New fixture solution for file I/O exercise.
tests/exercises/time-2-code/solution/solution.js New fixture solution for file I/O exercise.
tests/exercises/time-2-code/solution/solution.java New fixture solution for file I/O exercise.
tests/exercises/time-2-code/solution/solution.hs New fixture solution for file I/O exercise.
tests/exercises/time-2-code/solution/solution.cs New fixture solution for file I/O exercise.
tests/exercises/time-2-code/solution/solution.cpp New fixture solution for file I/O exercise.
tests/exercises/time-2-code/solution/solution.c New fixture solution for file I/O exercise.
tests/exercises/time-2-code/evaluation/plan.yml New evaluation plan using input/output files.
tests/exercises/output-files-custom-oracle/solution/wrong.ts New wrong-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/wrong.py New wrong-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/wrong.kt New wrong-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/wrong.js New wrong-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/wrong.java New wrong-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/wrong.cs New wrong-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/wrong.cpp New wrong-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/correct.ts New correct-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/correct.py New correct-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/correct.kt New correct-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/correct.js New correct-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/correct.java New correct-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/correct.cs New correct-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/solution/correct.cpp New correct-solution fixture for custom file oracle.
tests/exercises/output-files-custom-oracle/evaluation/suite.yaml New suite using output_files + custom oracle.
tests/exercises/output-files-custom-oracle/evaluation/evaluator.py Custom oracle implementation fixture.
tests/exercises/echo/solution/correct-files.ts Adds file-aware echo solution fixture.
tests/exercises/echo/solution/correct-files.sh Adds file-aware echo solution fixture.
tests/exercises/echo/solution/correct-files.py Adds file-aware echo solution fixture.
tests/exercises/echo/solution/correct-files.kt Adds file-aware echo solution fixture.
tests/exercises/echo/solution/correct-files.js Adds file-aware echo solution fixture.
tests/exercises/echo/solution/correct-files.java Adds file-aware echo solution fixture.
tests/exercises/echo/solution/correct-files.hs Adds file-aware echo solution fixture.
tests/exercises/echo/solution/correct-files.cs Adds file-aware echo solution fixture.
tests/exercises/echo/solution/correct-files.cpp Adds file-aware echo solution fixture.
tests/exercises/echo/solution/correct-files.c Adds file-aware echo solution fixture.
tests/exercises/echo/evaluation/plan.yaml Adds evaluation plan covering stdin-as-file.
tests/exercises/echo/evaluation/plan-dynamic.yaml Adds evaluation plan covering dynamic files.
tests/exercises/echo/evaluation/input.txt Adds resource fixture for !path stdin.
tested/testsuite.py Core suite model refactor for TextData and multi-file output.
tested/parsing.py New hook-chaining mechanism and updated fallback/ignore decorators.
tested/oracles/value.py Extends value extraction to support file output channels.
tested/oracles/text.py Extracts file oracle out; exposes text_options.
tested/oracles/file.py New multi-file-aware file oracle implementation.
tested/oracles/common.py Supports multi-result oracles + channel override reporting.
tested/oracles/init.py Routes file builtin oracle to new oracles.file.
tested/languages/preparation.py Updates before/after handling to TextData(content=...).
tested/languages/generation.py Updates readable input + linking for input_files/stdin-path.
tested/judge/planning.py Tracks input/output file conflicts; collects dynamic files per unit.
tested/judge/execution.py Materializes dynamic files into execution directories.
tested/judge/evaluation.py Supports per-file oracle results and new file-link messaging.
tested/dsl/translate_parser.py Adds !path, stdin file objects, input_files/output_files.
tested/dsl/schema.json Updates DSL schema for file data and output files.
tested/dsl/schema-strict.json Updates strict DSL schema for file data and output files.
tested/configs.py Updates @fallback_field decorator usage to new API.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@pdawyndt
Copy link
Contributor

pdawyndt commented Mar 7, 2026

I've generated test suites for all Time2Code's level 08 exercises. Those exercises make optional use of the file extensions that will be added to TESTed in this PR. Some test cases i) use multiple input/output files and ii) use the same file name with different content. The last feature is not supported by the original Python judge, so testing these exercises wasn't possible with the original judge. I'll test these level 08 exercises as soon as this PR has been put in production, and then work out the exercises from level 09 too.

@chvp chvp merged commit f8d89ea into master Mar 9, 2026
9 checks passed
@chvp chvp deleted the files-features branch March 9, 2026 09:01
chvp added a commit that referenced this pull request Mar 10, 2026
This reverts commit f8d89ea, reversing
changes made to bf4ae9a.
chvp added a commit that referenced this pull request Mar 10, 2026
Revert "Merge pull request #619 from dodona-edu/files-features"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants